Remote control signaling using audio watermarks

ABSTRACT

A system for using a watermark embedded in an audio signal to remotely control a device. Various devices such as toys, computers, and appliances, equipped with an appropriate detector, detect the hidden signals, which can trigger an action, or change a state of the device. The watermarks can be used with a “time gate” device, where detection of the watermark opens a time interval within which a user is allowed to perform an action, such as pressing a button, typing in an answer, turning a key in a lock, etc.

This application is continuation of commonly assigned co-pending U.S.patent application Ser. No. 10/794,520 filed on Mar. 5, 2004, which is adivisional of U.S. patent application Ser. No. 09/505,080 filed on Feb.16, 2000, now U.S. Pat. No. 6,737,957.

BACKGROUND OF THE INVENTION

The present invention relates to a method and apparatus for remotelycontrolling a device, such as a toy, lock, smart card, or homeappliance, via a control message that is imperceptibly embedded in anaudio signal, e.g., as a “watermark”. Moreover, the invention optionallyenables the device to be synchronized with the audio signal, forexample, so that the actions of a doll can be synchronized with achildren's television program.

Audio signals are ubiquitous, being broadcast over AM/FM radio, TV,public announcement systems, transmitted over telephone channels, orstored on cassette tapes, CDs, computer memories, etc. Therefore, it isconvenient to use audio channels or audio storage to transmit or storesome other information.

Audio watermarking, or embedded signaling, has recently emerged as atechnology for embedding auxiliary data imperceptibly in a host audiosignal. A basic feature of audio watermarking techniques is that theembedded signal is substantially imperceptible to a listener of the hostsignal. Furthermore, the audio-watermarks occupy the sametime/frequency/space domain as the host signal, so that they are notlost in standard audio signal processing, recording or transmissions,nor filtering and/or masking operations in a deliberate attack canremove them.

A primary proposed use of watermarking is in protecting intellectualproperty rights, e.g., through copy control, automatic broadcastmonitoring, ownership dispute resolution, Internet commerce tracking,etc. Alternative applications include auxiliary data embedding, such asthe song title and purchasing instructions, assurance of contentintegrity, proof of performance in TV and radio advertisements, audienceexposure monitoring, caller identification (authentication) in telephoneconversations, or generic covert communication.

Moreover, various schemes have been proposed for sending command andcontrol signals, or their equivalent, concurrently with audio signals.However, these schemes do not qualify as audio watermarking techniques.For example, in one proposed scheme, an “instructional signal” isinserted in a narrow frequency band set aside at the upper frequencyedge of the audio spectrum. However, this system does not qualify as awatermarking system since the host and the control signals occupydistinct frequency bands.

In another proposed scheme, a unique code describing an offer forproducts and services is transmitted by a TV program as an audible“beep”. There is no attempt to hide this beep, so this technique also isnot audio watermarking.

In yet another proposed scheme, information related to a TV game show isencoded in touch tones and broadcast in-band with an audio portion ofthe show. The touch-tones can be masked by the show's usual soundeffects, such as buzzers and beeps. This is substantially different fromthe watermarking approach, because it cannot simultaneously meet theinaudibility requirement and the requirement for the time domain overlapof a watermark and an arbitrary audio signal.

Accordingly, it would be desirable to provide a watermarking system forsending command and control signals concurrently with audio signals thatovercomes the disadvantages of the existing proposed schemes.

The system should use watermarking techniques to provide a hidden datachannel in an audio signal for providing short messages, such as deviceactivation commands, or remote control signals that can change the stateof a device.

The system should be compatible with existing watermaking techniques,such as those disclosed in U.S. Pat. No. 5,940,135 to Petrovic at al.,entitled “Apparatus and Method for Encoding and Decoding Information inAnalog Signals,” issued Aug. 17, 1999, and incorporated herein byreference.

The system should provide a hidden remote control signal as a watermarkwithin an audio signal for controlling various devices that detect thehidden signal.

The system should allow the remote control signal to be related to, orindependent of, content of the host audio signal. For related content,the system should optionally provide synchronization of the remotecontrol signal with the host audio signal content.

The system should use a watermark to define a time gate (window) duringwhich a device is enabled to receive a user input or perform a specifiedaction.

The system should provide a security mechanism to ensure that the timegate is defined only from a real-time broadcast audio signal, and notfrom a replay of the audio signal.

The system should improve the robustness and temporal resolution of awatermark, and provide a simplified watermark detector.

The system should provide synchronization of a watermark encoder anddecoder.

The present invention provides a system having the above and otheradvantages.

SUMMARY OF THE INVENTION

The present invention relates to a system for using a watermark embeddedin an audio signal to remotely control a device.

In particular, the system is compatible with existing audio-watermarkingtechnologies that use audio channels and/or audio storage to carryindependent data without interfering with the audio channel's originalpurpose. However, such a channel has much lower information capacitythan a modem channel, typically no more than about twenty bits persecond per audio channel. The invention uses this hidden data channelfor relatively short messages, such as device activation commands, orremote control signals that can change state of a device.

A remote control signal is hidden within an audio signal that isbroadcast over radio and TV, stored on CDs, DVD, tape or computermemory, played over speakers and/or transmitted over other audiochannels. Various devices such as toys, computers, and appliances, thatare equipped with an appropriate detector, detect the hidden signal totrigger an action, or change a state of the device. The device actioncan be completely unrelated to the ongoing audio content, and it canhave a number of different objectives, such as entertainment, education,sales, security, etc.

In one particular implementation, a “time gate” device is disclosed,where detection of the watermark opens a time interval within which auser is allowed to perform an action, such as pressing a button, typingin an answer, turning a key in a lock, etc. To prevent fraudulentactivation of a time gate, the time gate device can be further upgradedto react only to watermarks coming from live broadcasts, and not fromreplays from tapes or other storage devices.

In another implementation, detection of the watermark triggers anaction.

Additionally, techniques are presented for improving existingwatermarking technology in view of requirements for the proposedapplications. In particular, the invention provides improvements inrobustness of the watermark in the channels with acoustic propagation(e.g., propagation through air)—using delay hopping (watermarkingadjacent bits using distinct autocorrelation delays), robustnessimprovements using redundant watermarking, improvements in the timeresolution of the trigger feature, and simplifications of the detectordesign.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an audio watermarking process in accordance with thepresent invention.

FIG. 2 illustrates a system for remote control of a device, such as atoy, in synchronism with audio data, such as from a television program,in accordance with the present invention.

FIG. 3(a) illustrates a time gate defined by start and stop watermarksin accordance with the present invention.

FIG. 3(b) illustrates a time gate defined by a start watermark and afixed interval τ in accordance with the present invention.

FIG. 3(c) illustrates a time gate defined by a start watermark and amultiple N of a fixed interval τ′ in accordance with the presentinvention.

FIG. 4 illustrates a real-time time gate application in accordance withthe present invention.

FIG. 5 illustrates the use of countdown watermarks for defining a starttime of a desired action in accordance with the present invention.

FIG. 6 illustrates an autocorrelation modulation extractor based on signcorrelation in accordance with the present invention.

FIG. 7 illustrates an example of a bit error count versus a time shiftfor detecting a watermark in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to a system for using a watermark embeddedin an audio signal to remotely control a device.

FIG. 1 illustrates an audio watermarking process in accordance with thepresent invention.

At an encoding side 100, a watermark, i.e., an embedded signal, isinserted into an audio signal at an embeddor 105, using a key, which isa set of parameters that define the hiding process. The key may comprisea steganographic key. The composite signal that is output from theembeddor 105 can be recorded, transmitted over various channels, orprocessed in different ways, which usually includes corruption by noiseand distortion.

The composite signal is received at a decoding side 150, where theembedded signal (watermark) is retrieved from the composite signal in anextractor 155 with the help of the key used in the embedding process.

Various details regarding conventional signal processing techniques,such as compression, coding, error-correction, modulation, and the like,are not explicitly disclosed but their appropriate use will be evidentto those skilled in the art.

The embeddor 105 may provide the watermark in the audio signal usingvarious known watermarking techniques, including those discussed in theaforementioned U.S. Pat. No. 5,940,135, where the short-termautocorrelation function of the host audio signal is modulated in such away to match the embedded signal.

The key contains the information about the frequency band of the hostsignal used for hiding, the delay set used for autocorrelationcalculation and its change patterns, baseband symbol waveform, datapacket structure, scrambling key, etc. With this system, the extractor155 calculates the short-term autocorrelation using the same key, andregenerates the inserted message with the help of standard digitalsignaling techniques.

Typical bit rates of the embedded messages are low, ranging from a fewbits per second to a few tens of bits per second. For example, theInternational Federation of the Phonograph Industry required 20 bps perchannel for its review of watermarking technologies. Those skilled inthe art can appreciate that the increase in the bit rate brings reducedrobustness, increased audibility and/or increased complexity. Therefore,only relatively short messages can be hidden within an audio signal.This is quite suitable for intellectual property protection, where copycontrol information and/or content identification codes of less thanone-hundred bits are embedded.

However, the present invention proposes the use of audio watermarks forremote control of various devices, such as toys, locks, smart cards,appliances, etc., over standard audio channels, such as radio and TVbroadcasts, audio tapes, CDs, telephone channels, public addresssystems, etc. As an example, we will describe a system for remotecontrol of toys participating in a TV show, as illustrated on FIG. 2.

FIG. 2 illustrates a system for remote control of a device 290, such asa toy, in synchronism with audio data, such as from a children'stelevision program, in accordance with the present invention.

The control messages are inserted using audio watermarking atappropriate places of the show's audio track using an embeddor 205 and asynchronization device 210. The composite audio signal is optionallystored, e.g., on a tape 215, and subsequently broadcast via atransmitter 220 and antenna 225. While antennas 225 and 230 are shown asan illustration, any type of broadcast scheme can be used, includingdelivery via a terrestrial path, cable, optical fiber, and/or computernetwork. Moreover, broadcast of the composite signal to a largepopulation of receivers is not required, as the invention is alsosuitable for any transmission, including point-to-point transmissions,video-on-demand transmissions in a cable television network, and soforth. Moreover, the composite audio signal may be played back locallyat a user's home from a storage device such as a tape or disc.

A TV receiver 235 receives the signal via an antenna 230 and the audiosignal is played over a TV speaker 240. The toy 290, specially designedin accordance with the present invention, includes components 291,including a built-in microphone 292 for picking up (detecting) the audiosignal, a watermark extractor 294 for extracting the watermarks from thecomposite audio signal, and a control 295 that is responsive to thewatermarks for performing some action.

The toy 290 may optionally be hard-wired to the receiver usingappropriate jacks and wiring, in which case the microphone 292 is notneeded.

A motor 296, audio function 297, such as a speech synthesizer, andlights function 298 are responsive to the control 295. For example, ifthe toy 290 is a doll, the audio function 297 may play a prerecordedmessage in concert with the ongoing show. The motor 296 may cause thetoy's 290 arms, legs, head and/or mouth to move. The lights function 298may cause the toy's eyes to light up. This creates the appealingimpression that the toy is actually following the show together with thechildren, and that it participates.

Advantageously, no modification is required for the TV channel,including the TV signal storage equipment, satellite distributionchannels, broadcast equipment, and TV sets. Additional equipmentconsists only of an embeddor, specially designed for precise watermarkinsertion at a desired segment of the audio track, and a mass-produced,inexpensive detector incorporated in a suitably-designed toy andconnected to the toy controller 295. Also, note that the same toy 290can be activated by audio watermarks coming from any audio source, suchas an AM/FM radio broadcast, CD or tape player, or speakers wired to acomputer.

An important feature in the previous example is the synchronization ofthe action of a toy (or toys) with the ongoing show. To achieve this,the watermark should be embedded in the audio track segment immediatelypreceding the desired moment of the toy action, with a small allowancefor processing and propagation delays. This synchronizing feature can beuseful in many other practical applications.

Time Gate

For example, we will describe here a device suitable for watermark-basedactivation that we will call a “time gate” (e.g., window). In accordancewith the invention, the detection of a watermark causes the time gate toopen a time interval, during which a user is allowed to perform anaction. For example, during an interactive TV quiz show, the viewers mayparticipate by keying in their answers to a hand-held unit, while theplayers in the television studio prepare their answers. This can beachieved if the audio track has appropriately inserted watermarks, andif a time gate device synchronized to the watermarks controls access tothe hand-held unit.

Such a hand-held unit can have similar componentry as the toy 290 ofFIG. 2. The control 295 can be configured to send a message to the userto alert the user that the time gate has started or ended, the durationof the time gate, and the amount of time remaining in the time gate. Anoutput screen on the hand-held unit, such as a liquid-crystal display(LCD), may be appropriate for this purpose. Or, the hand-held unit maysend a signal back to the television receiver 235 to provide a displayon a TV or computer screen that informs the user of the informationprovided by the watermark (e.g., via a wired path, infra-red signal orthe like).

Alternative applications including an automated audio/video exam, wherethe time gate defines a period during which the user can enter responsesfor the exam, alertness monitoring, where the user is required toprovide an input during the time gate, TV coupon collection, whereelectronic coupons can be retrieved by a user during the time gate,remote control of a lock, where the lock can be opened or closed onlyduring the time gate, and so forth, are discussed below.

Three different designs of the time gate protocol are illustrated inFIGS. 3(a)-3(c).

FIG. 3(a) illustrates a time gate defined by start and stop watermarksin accordance with the present invention.

This design uses two distinct watermarks, including a start watermark305 and a stop watermark 310, to mark the desired beginning and the end,respectively, of the gated interval 315 bounded by T₁ and T₂. Note thatthe boundaries of the interval 315 are shown as occurring slightly afterthe end of the corresponding watermarks, due to processing andpropagation delays. This design can mark an arbitrary time interval thatis larger than the duration of the stop watermark 310, since theduration of the stop watermark 310 consumes a portion of the time gate315.

FIG. 3(b) illustrates a time gate defined by a start watermark and afixed interval τ in accordance with the present invention.

This time gate protocol design is simpler than that of FIG. 3(a) sinceit requires a single start watermark 330 before the beginning of themarked interval 335. The duration of the marked interval-(time gate), τ,is predefined, e.g., at the time the device, such as the toy 290 in FIG.2, is manufactured. However, the interval may be re-programmable (e.g.,by replacing a memory chip). This design is suitable if the user actionis simple, like pressing a button, or turning a key in a lock.

FIG. 3(c) illustrates a time gate defined by a start watermark and amultiple N of a fixed interval τ′ in accordance with the presentinvention.

This time gate design requires a single watermark 350, but still canmark a variable time interval. This is achieved by inserting an integernumber N into the watermark's data field, where the interval 360 is Ntimes a predefined time slot duration τ′.

The value N can be carried in a separate or same watermark as that ofthe control data, or even provided beforehand. A preferred solution isfor the same watermark 350 to include both N and the control data (forreasons of efficiency, potential for false action, etc.)

FIG. 4 illustrates a real-time time gate application in accordance withthe present invention.

Many time gate applications allow pre-processing of the audio track in asimilar manner as depicted in FIG. 1, e.g., for toy applications. Forexample, the time gate device can be used in an educational process tomonitor alertness of the student. In this case, a pre-recorded lecture(video/audio, or audio only) contains a watermarked content that opens atime gate interval after questions/instructions that require thestudent's response, such as pressing yes/no buttons on the time gatedevice. The device may score correct/incorrect answers for review by ateacher, but we expect major educational benefits by simply confirmingthat the student was paying attention to the audio signal.

Similarly, an advertisement company may poll viewers about an advertisedproduct, and engage their attention. Active participation of users isexpected to bring better recognition of products in a shorter time, andlower advertisement costs. In particular, as an encouragement forparticipation, an advertiser may offer discounts to those who bring thetime gate device to a specified retail store, where the user's answersto the poll questions are downloading from the device. This isequivalent to bringing cut-out coupons from a paper advertisement.Again, the watermarks are pre-stored in the audio track of a TV or radioadds.

However, in other applications, the embedding of the watermark shouldoccur in real-time, while the audio signal is being transmitted. Such acase occurs when the time gate is used, for instance, to control auser's access to a secure area or safe box, or to override otherrestrictions. For example, an innocuous public address systemannouncement or background music may be used with an electronic lock toallow a key to operate within the defined time interval. Similarly, auser may remotely control a door lock in the user's residence by callinghome over a telephone line, and speaking to the telephone answeringmachine (with its speaker on). The user's telephone set includes anencoder that embeds watermarks with control data into the user's voice.When the voice is received at a decoder with a watermark extractor at anelectronic door lock mechanism, the lock can be activated.

In a related example, background music or a public address announcementin a larger facility such as an office may be used as a host audiosignal for watermarked control data to remotely lock or unlock doors,filing cabinets and so forth without requiring re-wiring of thefacility.

With these applications, it is important that there is no detectablecommunication channel between the user and the controlled device, sothat potential attackers are oblivious to the control mechanism.Furthermore, if an attacker does learn the operational principle of thelock, he/she should not be able to gain access to the secure area byrecording and subsequently replaying the audio signal with the embeddedcontrol data.

Even if an attacker knows the operational principle of the lock, he/shecannot forge the message if he/she does not know the secret key used inthe embedding process, as shown in FIG. 1.

The real-time time gate system shown in FIG. 4 addresses these concerns.An encoding side 400 includes an audio source 405, an embeddor 410, aclock 415 and a control data encoder 420. A receiving side 470 thatreceives the real-time transmission of the composite audio signalincludes an extractor 475, a logic function 480, and a clock 485.

The embeddor 410 receives a continuous audio signal from the audiosource 405, and also receives a string of messages from the data encoder420. The messages include the control data, as well as timinginformation from the local clock 415, with possibly some additional data(e.g., number of slots for slotted time gate). The message embedding istriggered from outside, e.g., by an operator pressing a button on theembeddor 410. The message is inserted into the audio data stream using asecret key. In the real-time operation, the message is immediatelytransmitted, i.e., there is no recording of the audio signal except fora short buffering in the embeddor 410 necessary for the embeddingprocess (up to a few tens of milliseconds). The output composite signalis transmitted over a standard audio channel (telephone, radio, TV,public announcement system, etc). The receiver 470 detects the signaland passes it to the extractor 475, which detects the message and passesit to the logic function 480 for verification.

The logic function 480 compares the timing data from the incomingmessage to locally-generated timing data from the clock 485. If thematch is sufficiently close, the logic function 480 concludes that areal-time transmission has occurred, and the control information takenfrom the received message is passed, e.g., starting the time gate.Simultaneously, the local clock 485 may be adjusted (e.g., synchronizedwith the clock 415) if the discrepancy with the transmitter clock 415 iswithin predefined bounds. However, if the time discrepancy is too large(e.g., the local time is significantly after the transmitted time), itimplies that recording and play back may be taking place, so the messageis ignored.

If the local time is significantly before the transmitted time, thisimplies a significant mis-calibration, and a default mode can be invokedto ignore the message.

Clearly, potential attacks based on storing and replaying messages wouldintroduce large delays, certainly more than a few seconds. On the otherhand, propagation and processing delays are certainly less than asecond, so there should be a clear separation of the two cases. Also,typical clock devices can maintain a timing error well below a couple ofseconds for a quite long time, so that the time drift of local clockscan be also distinguished from the store-and-replay attack. Occasionalre-synchronization of the receiver clock with the transmitter's clockshould keep the timing mismatch between the send and receive clockswithin tight bounds indefinitely. This re-synchronization can beachieved using known techniques.

Watermark Design Considerations

The applications described above impose somewhat different sets ofrequirements on the audio watermark design than do the intellectualproperty protection applications. For example, most of the applicationsdescribed above imply an acoustic (free-space) propagation channel,while the property rights establishment is usually done on an electronicform of the audio signal. The use of an acoustic propagation channel isespecially challenging since it generates intersymbol interference dueto multipath, signals may be corrupted with background noise, and thedistortions in speakers and microphones are usually large compared toelectronic channel distortions.

Furthermore, some of the above applications, such as the toyapplications, require very inexpensive designs for the watermarkextractors. On the other hand, in some other applications, such asremote control of locks, it is equally important to have an inexpensiveembeddor as well, e.g., for the telephone set example. Finally, the timegate device requires careful consideration of timing tolerances toachieve the best possible resolution in time domain, as opposed totypical copyright protection applications, where the location of thewatermark within a signal is relatively unimportant.

With the above requirements in mind, the present inventors looked atdifferent watermarking technologies, and found that the best-suitedtechnology is the autocorrelation modulation (ACM) technique describedin the aforementioned U.S. Pat. No. 5,940,135. However, any othersuitable watermarking/data embedding technique may be used.

ACM features a simple design for the embeddor and extractor, highrobustness, large throughput, low probability of falses, good layeringcapability, and full inaudibility. However, the present inventionprovides improvements and the special selection of design parameters tooptimize ACM for the proposed applications. Herein, several techniquesare disclosed that can substantially improve the performance of ACM inremote control signaling using audio watermarks.

The acoustic-coupling environment, where a speaker broadcasts an audiosignal, and the detector captures it through a microphone, can beimproved by a special watermark design that is not addressed byconventional techniques. The main issue is the multipath propagationcaused by reflections of acoustic waves, which may introduce intersymbolinterference to the watermark detector. Standard techniques to fightintersymbol interference, such as adaptive equalization, are too costlyfor an inexpensive detector. An increase in the bit interval is helpful,but with obvious drawback in the reduction of the watermark bit rate.

To avoid intersymbol interference, we propose watermarking adjacent bitsusing distinct autocorrelation delays (“delay hopping”). In effect,distinct autocorrelation delays can be considered as distinct channels,with little interference between them. This aspect of the inventionincreases the watermark robustness, which is particularly useful in theacoustic-coupling environment, but may be used in other environments aswell. That is, if consecutive symbols are sent over distinct channels,they cannot cause intersymbol interference, regardless of the pulsebroadening caused by a multipath environment.

In a further aspect of the invention, robustness of the watermarking isimproved by first evaluating the masking ability of the host audiosignal before embedding the watermark. In a typical scenario, the deviceactivation, time gate opening, or other actions occur upon detection ofthe corresponding watermark. This means that the encoder should insertthe watermark immediately prior to the desired moment, taking intoaccount propagation and processing delays. However, the desired(candidate) watermark insertion interval may be unsuitable, for example,if it is mainly a silence. Accordingly, the watermark can be insertedbefore the desired instant of action, along with information forinforming the decoder about the delay between the watermark detectionand the desired action, which corresponds to the delay between the timesegment in which the watermark is embedded and the original desired timesegment.

There is a tradeoff between the flexibility in choosing the optimumwatermark insertion time and the amount of the payload (bits) assignedto the delay information.

FIG. 5 illustrates the use of countdown watermarks for defining a starttime of a desired action in accordance with the present invention.

In a further alternative embodiment, a string of watermarks 510 isembedded before the desired start of action (T₁). Each watermark, suchas watermarks 512, 514 and 516, has a countdown data field (n=2, 1, 0)that indicates the number of remaining watermarks before (T₁). Detectionof any of the watermarks in the string 510 allows the calculation of thedesired timing of the action. For example, if the countdown field of aparticular watermark contains the countdown field value n, and thewatermark duration is w seconds, than the desired action should beginn*w seconds after the particular watermark is detected, plus someadditional propagation and processing delay.

This provides improved robustness since the start time of the desiredaction is designated with redundance. Thus, even if all but one of thewatermarks are not received correctly, the start time will be still bedesignated.

FIG. 6 illustrates an autocorrelation modulation extractor based on signcorrelation in accordance with the present invention.

The simple decoder design described in the aforementioned U.S. Pat. No.5,940,135 includes a filter, followed by a delay line, correlator(multiplier), and an integrator. The output of the integrator representsa base-band watermark signal without normalization. When a binarymessage has been embedded, the output is positive at the decision momentif a “one” bit is embedded, or negative for a “zero” bit. In thisdesign, it is important to maintain a very precise delay in the delayline; small errors in the delay can bring significant distortion of thewatermark signal. Those skilled in the art know that precise delays arebest achieved by a digital delay line. This means that an A/D converteris necessary prior to the delay line, which add to the cost of thedecoder.

In accordance with the present invention, the decoder of U.S. Pat. No.5,940,135 can be further simplified as shown in FIG. 6 to meet evenlower cost demands, suitable for toy applications, without compromisingthe delay precision.

This decoder 600 includes a filter 610 that receives the compositesignal with the watermark, e.g., from a channel or a storage device, anda comparator 620 for comparing the filtered signal to ground 630. An ANDgate 640 receives the output of the comparator 620, and a local clocksignal. An XNOR (exclusive-NOR) gate 660 receives a direct output of theAND gate 640, as well as a shifted version of the output via shiftregister 650. The output of the XNOR gate 660 is provided to a counter670, which communicates with a logic function 680.

Thus, instead of multiplying the received signal with a delayed versionof itself, it is possible to detect the signal polarity, and thenperform an XNOR operation between the signal and a delayed version ofitself (i.e., if the signs are the same, the output is one, if opposite,the output is zero). Then, instead of integration, it is enough to runthe counter 670 at a clock rate that is much higher than the bit ratefor the duration of the watermark bit. If the count at the end of thebit interval is more than half the maximum count, then a “one” bit isdetected; otherwise a “zero” bit is detected. A synchronizing algorithm,residing in the logic block 680, determines the beginning and the end ofthe bit interval, and generates a reset signal for the counter at theend of each bit.

The above simplification shows that the comparator substitutes for anA/D converter, the XNOR gate 660 replaces a multiplicator, and thecounter 670 substitutes for an integrator. Moreover, the presentinventors have confirmed through experimentation that the proposedsimplification does not significantly reduce robustness.

FIG. 7 illustrates an example of a bit error count versus a time shiftfor detecting a watermark in accordance with the present invention.

In the time gate applications in particular, but in other triggeringapplications as well, it is important to achieve a good timing precisionin detecting the end of a watermark. In the case of watermarks describedin U.S. Pat. No. 5,940,135, this is equivalent to detecting the trailingedge of the last bit of the watermark bit stream. However, noise andchannel distortions can corrupt the trailing edge of any bit, or the bitas a whole. Therefore, it is necessary to take the watermark in itstotality to decide the most probable timing of the end of the watermark.

In typical digital watermarks described in U.S. Pat. No. 5,940,135,error correction codes are used to recover watermarks in the case whensome bits are corrupted. In this case, the watermark can be detected ata time interval that is slightly earlier than its optimal position, butwith a higher error count than in its optimal position. The presentinvention modifies this scheme propose to use this feature to furtheroptimize the watermark timing detection.

In accordance with the present invention, the decoder will attempt todetect the watermark with a starting and ending position that aredelayed slightly with respect to the timing position where initialdetection of the watermark occurred. The time shifts are very small withrespect to the watermark duration, typically of the order of 5% of thebit interval (T_(bit)) (each watermark contains tens or even hundreds ofbits). In each of these shifts, the decoder will continue to detect thesame watermark and monitor the bit error count over the duration of thewatermark. The same watermark is detectable even with slight timeshifts, which is the basis of the precise time resolution disclosedherein. The optimum timing is found where the error count is minimized.For multiple minimums, the optimum timing can be taken at the midpoint.

For example, in FIG. 7, the error count changes as the detecting time isshifted. Initially, with no time shift (i.e., at the originally-detectedposition), the watermark is detected with four bits in error (point710). However, as the detecting time is shifted in steps of ts (e.g.,ts=T_(bit)/20), the error count decreases (points 715 and 720), reachesa minimum for two shifts (points 725 and 730), and then increases(points 735, 740 and 745).

A minimum bit error (points 725 and 730) is reached for two consecutivesteps, and we conclude that the best estimate for a reference positionof the watermark (which can be, e.g., the ending time of the watermark)is at the mid-point of these two events, i.e., at (ts₃+ts₄)/2.Alternatively, either ts₃ or ts₄ could be selected. The resolution ofthe extractor's clock will govern the minimum possible time shift.

If only one minimum is found, the optimum position is taken at that timeshift position.

To reduce computations, it is possible to terminate the bit errorcalculations once a minimum has been detected and the bit error countbegins to rise (e.g., at point 735). Also, optionally, the bit errorcalculations can be terminated when a bit error count of zero is firstreached.

As an numeric example, assume the watermark message is two seconds long,and T_(bit)=0.08 sec. Let us say that watermark is first detected in theaudio signal starting at time 12.340 sec. and ending at time 14.340 sec.This corresponds to a zero time shift. Then, with ts1=0.08/20=0.004sec., the watermark is next detected starting at time 12.344 sec., andending at time 14.344 sec., and so forth. Note that the detectioninterval is 2 sec., and the position shift, and the resolution, is 0.004sec.

Moreover, for a strong watermark, the error count may reach zero or someother minimum value for several time shifts. In this case, the optimumtiming is again set in the middle of the time intervals with the minimumerrors.

Experiments show that this technique detects the watermark with theprecision of +/−20% of the bit interval. For example, for watermarksrunning at a 50 bit/sec. rate, this corresponds to a time resolution of+/−4 ms. This resolution corresponds to +/−1.4 meters of acousticpropagation delay, so it is adequate for the systems based on acousticcoupling.

Accordingly, it can be seen that the present invention provides a methodand apparatus with various advantages, including:

-   -   allows remote control of a device, such as a toy, lock, smart        card, or home appliance, via a control message that is        imperceptibly embedded in an audio signal as a watermark;    -   is compatible with, and builds upon, existing watermarking        techniques;    -   allows the remote control signal to be synchronized with the        audio content, such as to allow a toy to move in conjunction        with the audio track of a children's television program;    -   uses a watermark to define a time gate (window) during which a        device is enabled to receive a user input;    -   provides a security mechanism to ensure that the time gate is        defined only from a real-time broadcast audio signal, and not        from a replay of the audio signal;    -   provides a simplified watermark detector;    -   improves the robustness and temporal resolution of a watermark,        e.g., in acoustic propagation channels, by shifting the        detecting point based on an error count of the watermark; and    -   improves robustness by evaluating the masking ability of the        host audio signal when embedding the watermark, and inserting        the watermark in a suitable interval before the desired action,        along with information indicating the time shift, if necessary.

Although the invention has been described in connection with variousspecific embodiments, those skilled in the art will appreciate thatnumerous adaptations and modifications may be made thereto withoutdeparting from the spirit and scope of the invention as set forth in theclaims.

1. A method for controlling a user's interactions with a media content,comprising: receiving a media content that is embedded with at least afirst substantially imperceptible watermark at a user device; recoveringone or more of said embedded watermarks at said user device; commencinga time gate; and enabling a user action in accordance with saidrecovered one or more watermarks and said time gate.
 2. The method ofclaim 1, wherein said user action is not permitted outside a duration ofsaid time gate.
 3. The method of claim 1, wherein said user action isnot permitted within a duration of said time gate.
 4. The method ofclaim 1, wherein said user action is permitted after a fixed time fromthe recovery of said one or more watermarks.
 5. The method of claim 1,wherein said one or more watermarks comprise timing information and saiduser action is controlled in accordance with said timing information. 6.The method of claim 5, wherein said user action is permitted whenlocally generated time at the user device matches said timinginformation.
 7. The method of claim 5, wherein said user action ispermitted when locally generated time at the user device is within apredefined value of said timing information.
 8. The method of claim 1,wherein said one or more watermarks comprise a counter and said timegate is commenced in accordance with a predefined relationship to saidcounter.
 9. The method of claim 1, wherein said one or more watermarkscomprise duration information of said time gate.
 10. The method of claim1, wherein boundaries of said time gate are determined in accordancewith a start and a stop watermark.
 11. The method of claim 1, whereinsaid device is adapted to provide a response to said user actionoccurring within a duration of said time gate.
 12. The method of claim11, wherein said response to said user action comprises storinginformation related to said user action.
 13. The method of claim 11,wherein said response to a user action comprises relaying informationrelated to said user action to a different device.
 14. The method ofclaim 1, wherein said device is adapted to store informationcorresponding to said user action.
 15. The method of claim 1, whereinsaid device is adapted to relay information related to said recoveredone or more watermarks to a different device.
 16. The method of claim15, wherein said different device comprises one of a television or acomputer.
 17. A method for controlling a user's interactions with amedia content, comprising: embedding at least a first imperceptiblewatermark in said media content, said watermark being adapted to triggerthe commencement of a time gate once said watermark is received andrecovered at a user device such that a user action is enabled inaccordance with the recovered watermark and the time gate.
 18. Themethod of claim 17, wherein said user action is not permitted outside aduration of said time gate.
 19. The method of claim 17, wherein saiduser action is not permitted within a duration of said time gate. 20.The method of claim 17, wherein said user action is permitted after afixed time from the recovery of said watermark.
 21. The method of claim17, wherein said watermark comprises timing information and said useraction is controlled in accordance with said timing information.
 22. Themethod of claim 21, wherein said user action is permitted when locallygenerated time at the user device matches said timing information. 23.The method of claim 21, wherein said user action is permitted whenlocally generated time at the user device is within a predefined valueof said timing information.
 24. The method of claim 17, wherein saidwatermark comprises a counter and said time gate is commenced inaccordance with a predefined relationship to said counter.
 25. Themethod of claim 17, wherein said watermark comprises durationinformation of said time gate.
 26. The method of claim 17, whereinboundaries of said time gate are determined in accordance with a startand a stop watermark.
 27. The method of claim 17, wherein said device isadapted to provide a response to said user action occurring within aduration of said time gate.
 28. The method of claim 27, wherein saidresponse to said user action comprises storing information related tosaid user action.
 29. The method of claim 27, wherein said deviceresponse to a user action comprises relaying information related to saiduser action to a different device.
 30. The method of claim 17, whereinsaid device is adapted to store information corresponding to said useraction.
 31. The method of claim 17, wherein said device is adapted torelay information related to said recovered watermark to a differentdevice.
 32. The method of claim 31, wherein said different devicecomprises one of a television or a computer.
 33. A method for remotelycontrolling a device using a media content, comprising: receiving amedia content that is embedded with at least a first substantiallyimperceptible watermark at a user device; and recovering one or more ofsaid embedded watermarks at said user device; and triggering an actionat said user device within a time gate in accordance with the recoveredone or more watermarks.
 34. The method of claim 33, wherein said actioncomprises at least one of: movement of a mechanical component;displaying a message; storing a user response; or transmittinginformation to a different device.
 35. A method for remotely controllinga device using a media content, comprising: embedding a media contentwith at least a first substantially imperceptible watermark; andcommunicating said watermarked media content to said user device;wherein upon the reception of said watermarked media content by saiduser device one or more of said watermarks is recovered, and an actionis triggered within a time gate in accordance with said recovered one ormore watermark.
 36. The method of claim 35, wherein said actioncomprises at least one of: movement of a mechanical component;displaying a message; storing a user response; or transmittinginformation to a different device.